Fingerprints, or why your alert storm is one bug

An exception arrives about twelve hundred times a second across our ingest fleet. The first real decision we make about each one is whether it is new.

Get it wrong in one direction and a single bad deploy pages a team eleven thousand times. Get it wrong in the other and two unrelated failures share an issue, one of them gets marked resolved, and the other disappears.

What goes into the fingerprint

By default, three things: the exception type, the top frames of the stack that belong to your code, and the module they live in. Everything else is deliberately left out.

• Line numbers are out. Adding a comment above a function would otherwise open a new issue on every deploy.
• Framework and vendor frames are out. Two different bugs meeting in the same ORM method are two bugs.
• The message is out unless it has no variables in it. Order 58213 not found and Order 61904 not found are the same bug; we strip the number and keep the shape.

The three times to override it

Set fingerprint on the event yourself when the default cannot know what you know.

Artisan::configureScope(function ($scope) {
    $scope->setFingerprint(['carrier-timeout', $carrier]);
});

• One wrapper, many causes. A generic IntegrationException around eleven carriers should be eleven issues, split by the tag that names the carrier.
• Many wrappers, one cause. A dependency that fails three different ways when a single credential expires is one issue on a Tuesday morning.
• Noise you have decided to watch as a rate. Grouping every 429 into one issue turns a storm into a graph you can put a threshold on.

When we get it wrong

Two buttons, both reversible. Merge takes issues that should have been one, and keeps the incoming events flowing to the survivor. Split takes an issue apart along a tag, and re-fingerprints the events already in it, which is why it can take a minute on a large one.

Both write to the audit log, and both leave a note on the issue saying who did it and when, because six weeks later the interesting question is always why an issue looks the way it does.